A Toolset for Supporting Iterative Human Automation: Interaction in Design

The addition of automation has greatly extended humans' capability to accomplish tasks, including those that are difficult, complex and safety critical. The majority of Human - Automation Interact~on (HAl) results in more efficient and safe operations, ho,,:,ever ~ertain un~~pected a~tomatlon behaviors or "automation surprises" can be frustrating and, In certain safety critical operations (e.g. transport~tion, manufacturing control, medicine), may result in injuries or. the loss of life.. (Mellor, 1994; Leveson, 1995; FAA, 1995; BASI, 1998; Sheridan, 2002). This pap~r describes ~he development of a design tool that enables on the rapid development and evaluation. of automat~on prototypes. The ultimate goal of the work is to provide a design platform upon which automation surprise vulnerability analyses can be integrated

Best Practices for Evaluating Flight Deck Interfaces for Transport Category Aircraft with Particular Relevance to Issues of Attention, Awareness, and Understanding CAST SE-210 Output 2 Report 6 of 6

Attention, awareness, and understanding of the flight crew are a critical contributor to safety and the flight deck plays a critical role in supporting these cognitive functions. Changes to the flight deck need to be evaluated for whether the changed device provides adequate support for these functions. This report describes a set of diverse evaluation methods. The report recommends designing the interface-evaluation to span the phases of the device development, from early to late, and it provides methods appropriate at each phase. It describes the various ways in which an interface or interface component can fail to support awareness as potential issues to be assessed in evaluation. It summarizes appropriate methods to evaluate different issues concerning inadequate support for these functions, throughout the phases of development

Some Challenges in the Design of Human-Automation Interaction for Safety-Critical Systems

Increasing amounts of automation are being introduced to safety-critical domains. While the introduction of automation has led to an overall increase in reliability and improved safety, it has also introduced a class of failure modes, and new challenges in risk assessment for the new systems, particularly in the assessment of rare events resulting from complex inter-related factors. Designing successful human-automation systems is challenging, and the challenges go beyond good interface development (e.g., Roth, Malin, & Schreckenghost 1997; Christoffersen & Woods, 2002). Human-automation design is particularly challenging when the underlying automation technology generates behavior that is difficult for the user to anticipate or understand. These challenges have been recognized in several safety-critical domains, and have resulted in increased efforts to develop training, procedures, regulations and guidance material (CAST, 2008, IAEA, 2001, FAA, 2013, ICAO, 2012). This paper points to the continuing need for new methods to describe and characterize the operational environment within which new automation concepts are being presented. We will describe challenges to the successful development and evaluation of human-automation systems in safety-critical domains, and describe some approaches that could be used to address these challenges. We will draw from experience with the aviation, spaceflight and nuclear power domains

Evaluating Models of Human Performance: Safety-Critical Systems Applications

This presentation is part of panel discussion on Evaluating Models of Human Performance. The purpose of this panel is to discuss the increasing use of models in the world today and specifically focus on how to describe and evaluate models of human performance. My presentation will focus on discussions of generating distributions of performance, and the evaluation of different strategies for humans performing tasks with mixed initiative (Human-Automation) systems. I will also discuss issues with how to provide Human Performance modeling data to support decisions on acceptability and tradeoffs in the design of safety critical systems. I will conclude with challenges for the future

Airplane Capabilities: Translating Non-Normal Information for Operational Decision-Making

We consider how a jet transport airplane interface supports the flight crew in managing airplane system failures (or non-normals) for continued safe flight and landing. The existing state of the art starts with a list of airplane system component failures and asks the flight crew to determine, with the help of non-normal procedures, the operational consequences of those failures. As airplane systems become more complex and interconnected, the flight crew's ability to determine operational consequences will become inadequate. We describe an approach that attempts to translate airplane system failures directly into airplane "capabilities," which is a set of basic airplane functions, such as the ability to stop after landing. This paper describes the overall framework for supporting flight crews in operational decision making and the initial efforts to develop a language and display concepts

The Role of Alerting System Failures in Loss of Control Accidents CAST SE-210 Output 2

This report is part of a series of reports that address flight deck design and evaluation, written as a response to loss of control accidents. In particular, this activity is directed at failures in airplane state awareness in which the pilot loses awareness of the airplane's energy state or attitude and enters an upset condition. In a report by the Commercial Aviation Safety Team, an analysis of accidents and incidents related to loss of airplane state awareness determined that hazard alerting was not effective in producing the appropriate pilot response to a hazard (CAST, 2014). In the current report, we take a detailed look at 28 airplane state awareness accidents and incidents to determine how well the hazard alerting worked. We describe a five-step integrated alerting-to-recovery sequence that prescribes how hazard alerting should lead to effective flight crew actions for managing the hazard. Then, for each hazard in each of the 28 events, we determine if that sequence failed and, if so, how it failed. The results show that there was an alerting failure in every one of the 28 safety events, and that the most frequent failure (20/28) was tied to the flight crew not orienting to (not being aware of) the hazard. The discussion section summarizes findings and identifies alerting issues that are being addressed and issues that are not currently being addressed. We identify a few recent upgrades that have addressed certain alerting failures. Two of these upgrades address alerting design, but one response to the safety events is to upgrade training for approach to stall and stall recovery. We also describe issues that are not being addressed adequately: better alert integration for flight path management types of hazards, airplanes in the fleet that do not meet the current alerting regulations, a lack of innovation for addressing cases of channelized attention, and existing vulnerabilities in managing data validity

A Summary of Results from Technologies for Aircraft State Awareness Safety Enhancement 210 Output 2

In 2014, the Commercial Aviation Safety Team produced a report describing the results of an analysis intended to understand and mitigate airplane incidents and accidents associated with flight crew loss of attitude or energy state awareness. That report described several "Safety Enhancements" including a new category of "Research Safety Enhancements". This paper focuses on Safety Enhancement (SE) 210 Output 2, investigating improvements in design methods and guidelines to "assess flight crew performance in situations associated with loss of energy and/or attitude state awareness"

Autonomous Systems Taxonomy

The purpose of this taxonomy is to provide common definitions and a functional decomposition of the technology that is required for NASA's autonomous systems. The taxonomy serves as a framework for: (1) assessing the state of NASA's autonomous systems capability (workforce, technology, etc.) and (2) assessing the state of the art in autonomy technology

Benefits of Matching Domain Structure for Planning Software: The Right Stuff

We investigated the role of domain structure in software design. We compared 2 planning applications, for a Mission Control group (International Space Station), and measured users speed and accuracy. Based on our needs analysis, we identified domain structure and used this to develop new prototype software that matched domain structure better than the legacy system. We took a high-fidelity analog of the natural task into the laboratory and found (large) periformance differences, favoring the system that matched domain structure. Our task design enabled us to attribute better periormance to better match of domain structure. We ran through the whole development cycle, in miniature, from needs analysis through design, development, and evaluation. Doing so enabled inferences not just about the particular systems compared, but also provided evidence for the viability of the design process (particularly needs analysis) that we are exploring